Identifying Threats to Validity and Control Actions in the Planning Stages of Controlled Experiments

During the planning phase of an experiment, we must identify threats to validity in order to assess their impact over the analyzed data. Furthermore, actions to address these threats must be defined, if possible, so that we can minimize their impact over the experiment’s results. In this paper, we propose the Threats to Validity Assistant (TVA), which is a tool to assist novice researchers in identifying and addressing threats to validity of controlled experiments in Software Engineering. The TVA was evaluated through a controlled experiment which aimed at measuring the effectiveness and efficiency in identifying threats to validity and actions to address them. Sixteen subjects were asked to identify threats to validity and control actions related to an empirical study reported in literature. Our results indicate that the subjects who used TVA were significantly more effective and efficient than the subjects who did not use it.